Power loss siren

ABSTRACT

In an embodiment, a power management system includes a detection unit configured to detect a power interruption to a power supply. The system further includes a communication interface configured to, in response to the detected power interruption, provide a message regarding the detected power interruption. In response to the detected power interruption, a computer network switch provides notifications to a plurality of servers connected to the switch to allow the plurality of servers to prepare for a loss in power.

BACKGROUND OF THE INVENTION

A data center supports a variety of IT operations such as cloudcomputing and large-scale services such as application or data services.A data center houses racks of computer servers, which provide processingand data storage functionalities. The processing and data storagefunctionalities are implemented by processing, telecommunication, andnetworking equipment such as processors, switches, and routers in theserver racks.

Service disruptions such as power failures may result in consequencesfor rack-mounted devices. For example, there may be disruptions toservices due to a power failure. Conventionally, a data center isconnected to an AC utility grid. A server rack in a typical data centerimplemented according to the Open Compute Project converts the AC powerto DC power via power supply modules for use by rack-mounted devices.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 illustrates an example of a power distribution flow from a powergenerator to a server rack.

FIG. 2 is a block diagram of a system for distributing power within aserver rack according to an embodiment.

FIG. 3 is a circuit diagram of a power supply circuit of a power supplyunit in a power shelf according to an embodiment.

FIG. 4 is a block diagram illustrating an embodiment of a server rack inwhich a power management system is provided.

FIG. 5 is a block diagram illustrating an embodiment of a rack monitorfor providing a power loss siren.

FIG. 6 is a flow chart illustrating an embodiment of a process forproviding a power loss siren.

FIG. 7A is a front view of an example of a physical rack monitor.

FIG. 7B is a perspective view of an example of a physical rack monitor.

FIG. 8 shows an example of a physical rack monitor mounted on a serverrack.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications, andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

Providing a notification associated with a power loss is disclosed. Apower loss siren may include a notification of imminent power loss. Thepower loss siren is implemented by a power management system thatnotifies data center components before power to the components issuspended. Conventionally, server components do not receive warningsbefore a power loss, which is disruptive to services if the componentsare unprepared. The disclosed power loss siren mitigates some of therisk of downtime due to power disturbances by allowing data centercomponents to prepare for the power loss. As more fully described below,a server rack includes a detection unit (such as a power supply unit), acomputer network switch to manage components within a server rack andcommunications between racks, power supplies to service rack components,and servers. In various embodiments, a power management system includesa detection unit configured to detect a power interruption to a powersupply. For example, if there is an outage in a utility grid, power tothe power supply of the server rack may be interrupted. The powermanagement system also includes a communication interface configured to,in response to the detected power interruption, provide a message (e.g.,to a computer network switch) regarding the detected power interruption.In response to the detected power interruption, the network switchprovides notifications to servers connected to the computer networkswitch to allow the servers to prepare for a loss in power.

FIG. 1 illustrates an example of a power distribution flow from a powergenerator to a server rack. Electricity is generated from a powerstation (e.g., primary utility 102) and delivered to a consumer (in thisexample, a data center) according to this example distribution flow. Forexample, an electric utility may generate power in a variety of ways(hydroelectricity, wind turbines, solar panels, etc.). The electricalenergy is then transmitted from the power plant to a substation and thento consumers. The transmission involves stepping up the power andstepping down the power at various stages. In this example flow, thefocus is on the later stages of electric power delivery in whichelectricity is delivered from a transmission system to consumers. Here,transformer 104 lowers the voltage to a voltage suitable for mainswitchboard 106 of a building. For example, a local electrical supplysupplies a main switchboard in the building.

Main switchboard 106 is a device that directs electricity from a sourceof supply to smaller regions of usage. The main switchboard providespower feeds to other distribution/sub-distribution boards in thebuilding. For example, main switchboard 106 services an entire buildingby directing power supplied by transformer 104 to distributionswitchboard 108. Each distribution switchboard 108 services a smallerregion within the building such as a floor of the building. A maindistribution board may be located on a floor, while a sub-distributionboard provides a feed into a server room. Individual power feeds can beprovided for rack power distribution units. In some embodiments, eachmain switchboard has a standby generator that provides power in theevent of utility outage.

Distribution switchboard 108 delivers power to a reactor power panel110. Reactor power panel 110 can be a custom device that deliversthree-phase power to the server cabinet level. For example, the reactorpower panel can improve efficiency by reducing short circuit current,correct leading power factor, reduce current harmonic distortion, andthe like. Although two RPPs are shown here (in a dual and a singleconfiguration), addition RPPs may be provided. For example, in somesystems each of the RPPs represent a group of 12 RPPs. In someembodiments, power is conducted via a busway (also known as bus bar orbus duct) to a tap box. A tap box facilitates power restoration to abuilding or portion of building if there is an outage. For example, itcan be connected to an automatic or manual transfer switch in the powerdistribution network, working in cooperation with the switch and abackup generator to restore power if there is an outage. A tap box canbe electrically coupled to rack power distribution unit (PDU), and isconfigured to supply power to the rack PDU. The rack PDU is adistribution unit corresponding to a server rack such as the one shownin FIG. 4. Rack PDU delivers power to one or more power shelves in theserver rack.

A power shelf contains power supply units that use the load-balancingbenefits of the three-phase input from the power grid to deliver powerto components in a server rack. The power shelf can rectify each phase(e.g., a hot wire and neutral wire) in parallel to provide DC voltagethrough a single DC bus pair (e.g., a positive and negative DC lowvoltage terminations). The DC bus pair can provide power to rack-mountedequipment in a “power zone.” In some embodiments, a power shelf includesthree pairs of power supply units (PSUs) and backup battery units(BBUs). Each pair is referred to as a “power module.” Each PSU in eachpair can receive one of the three AC phases from the AC power source(here, via rack PDU). This enables the power supply system to draw fromthe three-phase power in a way that the power never or rarely falls tozero, meaning that the load is the same at any instant, ensuring asubstantially even balance of the input AC phases. The three powermodules are coupled in parallel to a DC bus to provide power torack-mounted equipment within the server rack. Here, for example, poweris delivered by the power shelf to a power supply unit, which in turndelivers power to a server and server components such as a CPU. Anexample of a power shelf is further described with respect to FIG. 4.

The power distribution can have a variety of example voltages. Thesevoltages are merely exemplary and not intended to be limiting. Forexample, main switchboard 106 converts 2.5 MW to 1.25 MW fordistribution switchboard 108. Distribution switchboard 108 then convertsthe voltage to around 155-189 kW for the reactor power panel 110. By thetime the power reaches the tap box, the voltage is lowered to 12.6 kW.In this example, a power shelf takes voltage at 6.6 kW, and each powersupply unit in the power shelf supplies 3.2 kW of power. A typical twosocket compute server consumes 300-400 W of power, and each CPU consumes200 W or less of power, depending on generation. These numbers canchange based on hardware generation and configuration. Storage devicesmay have different power profiles.

Redundancies built into the power distribution flow protects againstpower outages that impact server uptime. In conventional systems, onearea that does not have much built-in redundancy is downstream of thedistribution switchboard 108. Outages downstream of the distributionswitchboard directly impacts server availability and uptime. The powermanagement system disclosed here reduces the impact of power outages bybroadcasting an AC power outage notification within a server rack. Thisincreases resiliency to unplanned power outages in data centers, forexample preventing service interruptions. The performance of the servernetwork is thus improved.

The following figure shows an example of how power is distributed withina server rack.

FIG. 2 is a block diagram of a system for distributing power within aserver rack according to an embodiment. The system includes power supplyunit (PSU) 210 and battery back-up unit (BBU) 280. PSU 210 is usuallypowered by AC input 220 from a rack power distribution unit (e.g., rackPDU 116 of FIG. 1). In case the AC input is unable to power the PSU suchas during a power outage, BBU 280 provides backup power for a relativelybrief time when the AC utility grid is unable to deliver power. In otherwords, BBUs can extend power for some time before rack-mounted devicesperceive a power outage. However, BBUs can provide back-up power foronly a limited time, and data may be lost for example if a server is inthe middle of an operation when power is lost. The power loss sirentechniques described here proactively respond to power disruptions bywarning server hosts of imminent power loss.

In this example, PSU 210 supplies power at two different DC voltages: DCbus, (here, 12.5V) and auxiliary power (here, 54V). The two different DCvoltages power components that use different voltages. The DC bus isserviced by the PSU as follows. AC input 220 is received from a rackPDU. The signal can be processed by an AC power circuit including EMIfilter 202, rectifier 204, and power factor corrector 206. Thesecomponents are configured to remove EMI noise/interference, rectify anAC current to DC voltage, and provide power correction to the rectifiedsignal, and are further described with respect to FIG. 3. Theconditioned signal is output to bulk capacitor 208. The bulk capacitoris configured to provide smooth voltage transitions during AC outagesand AC restores, and holds electric charge at the bulk voltage terminalsuch that a substantially constant DC voltage is maintained. The poweris then converted by each of DC/DC converter 212 and DC/DC converter 218to a desired voltage to power the components of a server rack in whichthe PSU is provided.

When rack PDU is operating normally, e.g., providing sufficient power toPSU 210, BBU 280 is typically not powering the PSU. Instead, the BBU ischarged by charger 216 so that, in the case of a power outage, the BBUcan be used to power the PSU. When there is a power outage, BBU 280services PSU 210 by providing DC input to DC/DC converter 214. Theconverted signal may be further stepped up or down by DC/DC converter212 or DC/DC converter 218 to a desired voltage.

The voltages shown for the DC/DC converters 212, 214, and 218, charger216, and output voltages corresponding to the DC bus and AUX are merelyexemplary and not intended to be limiting.

The following figure is an example of circuit implementing the powerdistribution flow described with respect to FIG. 2.

FIG. 3 is a circuit diagram of a power supply circuit of a power supplyunit in a power shelf according to an embodiment. For example, a powersystem can include multiple power shelves, e.g., power shelf 404A and404B of FIG. 4, within a server rack. Each power shelf includes at leasttwo PSUs (in a single phase solution) or includes at least three PSUs(in a three-phase solution). In various embodiments, each PSU includesthe power supply circuit 300 disclosed herein.

The power supply circuit 300 includes an AC power circuit 302 and a DCpower circuit 304. The AC power circuit 302 receives an AC phase 306 ofAC power and a neutral line 308 from a power grid. For example, the ACphase 306 can provide 277V AC power. The AC phase 306 can be one ofthree AC phases provided through the power grid. The AC phase 306 andthe neutral line 308 can feed into a rectifier circuit 310 to rectifythe AC power into or approximately into DC voltage. The rectifiercircuit 310 may further include an inrush control protection circuit(not shown) to prevent electromagnetic interferences and electricdischarges of the AC phase 306.

The rectifier circuit 310 can be coupled to a PFC module 312. The PFCmodule 312 provides power correction to the rectified output of therectifier circuit 310. For example, the PFC module 312 can removeeffects of leakage inductance or reactance that limits the outputcurrent of the rectifier circuit 310 and/or adjust the power factor andthe iTHD (current total harmonic distortion). The PFC module 312 canoutput a DC voltage (hereinafter the “bulk voltage”) to a bulk voltageterminal 314. That is, the AC power circuit 302 receives an electricalinput at the AC phase 306 and provides a DC electrical output at thebulk voltage terminal 314.

A bulk capacitor 316 can be coupled between the bulk voltage terminal314 and a first negative return terminal 318A. The bulk capacitor 316can hold electric charge at the bulk voltage terminal 314 such that asubstantially constant DC voltage (e.g., around 440 volts) can bemaintained at the bulk voltage terminal 314 even without the PFC module312 being operational. The bulk capacitor 316 can provide smooth voltagetransitions during AC outages and AC restores.

The bulk voltage terminal 314 is coupled to a DC-DC converter 319 as theinput to the DC-DC converter. For example, the bulk voltage terminal 314can be connected via a high resolution current share bus. The highresolution current share bus can be used to provide high output currentshare accuracy. In some embodiments, the converters drawing current fromthe bulk voltage terminal 314 share the output current very precisely.In this way, the respective input AC RMS currents would be well balancedand so, the three three-phase input AC currents would have similarvalues and therefore be balanced. The precise current sharing enablesthe power supply circuit 300 to maintain the same load over the threephases of a three-phase AC input from the power grid.

The DC-DC converter 319 can down-step the DC voltage to a lower voltage.The lower voltage is provided throughout a power zone associated withthe PSU that the power supply circuit 300 is part of. The lower voltageis provided to the power zone via a DC bus 320. The DC bus 320, forexample, provides a 12V or a 12.5V DC voltage relative to a secondnegative return terminal 318B. The DC bus 320 can create a common DCvoltage used to power the system in a server rack supported by the powersupply circuit 300. For example, the DC bus 320 can also connect inparallel to other PSU and BBU pairs. This parallel connection isadvantageous since if one of the parallel connections suddenly fails,the remaining two would not be affected by the failure and would be ableto carry on the full power required by the power system.

In various embodiments, the power supply circuit 300 represents one ofthree power modules in a power shelf, where each power module receives asingle phase input. The DC bus 320 is connected in parallel to otheroutput voltages other than the output of the DC-DC converter 319. Thisconnection enables a redundant power shelf solution. A power shelf withthese parallel PSU bus connections generate enough power to sustain thesystem load. In other embodiments, the DC bus 320 may instead byconnected to one power module with a three-phase AC input with a secondpower module connected in parallel for redundancy. These alternativeembodiments would cost more because a (1+1) redundant scheme and athree-phase input power supply normally costs more than a single-phaseinput power supply. Further, the three-phase input is normally lessefficient than single phase AC inputs.

The DC-DC converter 319 can be coupled to an isolated transceiver 322that guards against large negative return-to-negative returndifferentials. The isolated transceiver 322 can be coupled to a thirdnegative return terminal 318C. The negative return terminals areisolated from each other to avoid return loop currents and to limitdifferential and common mode noise. The negative return isolation alsoavoids load return currents that may be flowing through high powercurrent return paths (e.g., taking the wrong route through signal returnpaths when the negative returns are not isolated). The power supplysystem can share the same eventual negative return termination at thepower shelf. If the isolated transceiver 322's negative return isisolated at the source, return loop currents would be impossible. Anisolation table 323 lists the voltage isolation levels of the (isolated)converters built-in inside the power supply circuit 300. Theseisolations levels can be sized by those who design the isolated powerconverters.

The isolated transceiver 322 may output a digital line 325 for externalcommunication (e.g., health status reporting) with devices outside ofthe power shelf or for internal communication between instances of thepower supply circuit 300 in the same power shelf (e.g., a power shelfwith three power supply circuit 300 each corresponding to a PSU and BBUpair).

The DC power circuit 304 is coupled to a battery backup unit (BBU) 324(shown with dotted lines to indicate that the BBU 324 is not part of thepower supply circuit 300), and includes a current feed converter 326 anda battery charger 328. For example, the DC power circuit 304 may includean electrical interface to couple with the BBU 324. The BBU 324 caninclude a set of batteries connected in series. For example, the set ofbatteries can provide up to 37V to 54V of DC differential relative tothe second negative return terminal 318B. The BBU 324 can detachablyconnect to the power supply circuit 300 of the PSU. The DC voltageoutput of the BBU 324 is connected to the current feed converter 326.The current feed converter 326 steps up the DC voltage of the BBU 324 tothe constant DC voltage at the bulk voltage terminal 314. In someembodiments, an asymmetric conductance component 329 (e.g., a diode) iscoupled in between the current feed converter 326 and the bulk voltageterminal 314. The asymmetric conductance component 329 ensures thatcurrent flows only in the direction from the current feed converter 326to the bulk voltage terminal 314 and not vice versa.

The BBU 324 supports powering of the DC bus 320 in case of AC power gridoutage. The current feed converter 326 can be configured to dischargethe BBU 324 for a set amount of time during an outage (e.g., a 90seconds timeout), where the BBU 324 shuts down when the backup sequencereaches the set amount of time. In some embodiments, the BBU 324'svoltage is only connected within the power supply circuit 300representing its corresponding PSU, forming a pair. Each instance of theBBU 324 is independent against remaining BBUs installed in the powershelf. For instance, the BBU 324 extends the hold-up time of itscorresponding PSU from a usual 20 milli-seconds (e.g., by a capacitor)to a full 90 seconds or more. The 20 milli-seconds may correspond to afull cycle of the input AC sinusoidal voltage at 50 HZ. A power losssiren can be raised before the duration by which the BBU extends thehold-up time ends. The power loss siren prepares components powered bythe power shelf for a power outage.

The battery charger 328 provides suitable voltage to charge the BBU 324when the AC power circuit 302 is operational (e.g., the AC phase 306 isproviding power and the PFC is functioning). The battery charger 328,for example, can be an isolated step-down converter connected betweenthe bulk voltage terminal 314 and the BBU 324.

The power supply circuit 300 can include a PSU logic module 330. The PSUlogic module 330 can be an application specific integrated circuit(ASIC), a field programmable gate array (FPGA), a controller, aprocessor, or other logical units capable of computation. The PSU logicmodule 330 is configured to control the current passing through thecurrent feed converter 326. The PSU logic module 330 can be coupled tothe BBU 324 (e.g., through the electrical interface of the DC powercircuit 304 or through another electrical interface) to monitor powercondition of the BBU 324 and to react on signals coming from the BBU324. For example, the BBU logic can determine an internal failure (andother electrical and environmental conditions) and report it to the PSUlogic module 330 of same pair. For example, the BBU logic cancommunicate with the PSU logic module 330 through a digital bus 331. Thedigital bus 331 may be used for communication between the BBU and thePSU logic module 330 as well as for reporting state of healthinformation from the BBU to the PSU logic module 330. The PSU logicmodule 330 can also determine when the BBU 324 is over-charged,over-discharged, over-heated, and etc. Based on the power condition ofthe BBU 324, the PSU logic module 330 can assess in real-time the healthof the pair (PSU and BBU), and stop either charge or discharge asneeded, under several protection modes. For example, when performing abattery test, the PSU logic module 330 can discharge the BBU 324 atconstant current, constant power, or at real-time system power. The PSUlogic module 330 can also stop a BBU test at anytime as needed or whenconsidered necessary per diagnostic by the PSU logic module 330 or theBBU logic. Similarly, the PSU logic module 330 can discharge the BBU 324for backup power discharge in case of AC power grid outage.

The power supply circuit 300 can further include a droop share converter332. The droop share converter 332 can be a low-power isolated droopshare converter. The droop share converter 332 is coupled in parallel tothe respective DC-DC converters (e.g., the DC-DC converter 319) in theremaining power modules in the power shelf. The droop share converter332 is a DC to DC converter that steps down the bulk voltage to a DCvoltage different from the DC voltage at the DC bus 320 (e.g., alsoknown as “AUX voltage”, and can be typically 54V DC). Droop shareconverters enable load sharing in an array of converters. For example,the droop share converter 332 can artificially increase the outputimpedance to force the currents through the DC-DC converter 319 and thedroop share converter 332 to be equal, without actually dissipating anypower. This may be accomplished, for example, by an error signal whichis interjected into the control loop of the droop share converter 332.

In various embodiments, the voltage at the DC bus 320 is protected fromAC power grid outage, because the DC-DC converter 319 is fed from theBULK voltage terminal 314 within the same PSU. The AUX voltage can beconveniently used to power rack switching gears, and any other auxiliaryrack gears that may need continuous reliable power. Switching gears cannormally be powered by one power supply with an additional power supplyfor redundancy (e.g., a (1+1) redundant power setup). Hence, normallyswitching gears can use two power supplies in parallel, while one powersupply is enough to sustain the full load. At the rack level, the twopower supplies can be fed by the AUX voltage coming from a first powershelf installed in the rack, and the AUX voltage coming from a secondpower shelf in the rack. This would be a full redundant layout, wherethere are two independent AUX sources, each one with three pairs of PSUand BBU (e.g., a (2+1) redundant setup as explained above), and twoindependent power supplies installed in the switching gear.

Each of the modules/circuits/components may operate individually andindependently of other modules, circuits, or components. Some or all ofthe circuits, components, and/or modules may be combined as onecomponent or module. A single module/circuit/component may be dividedinto sub-modules, each sub-module performing a separate electricaltransformation or transformations of the original singlemodule/circuit/component.

The following figure is an example of a server rack with componentscorresponding to the intra-rack distribution flow of FIGS. 2 and 3.

FIG. 4 is a block diagram illustrating an embodiment of a server rack inwhich a power management system is provided. This example shows a singlerack column. In various embodiments, rack columns may be grouped intoseveral (e.g., three rack columns), and the group of rack columns mayshare one or more components.

Rack column 400 includes a computer network switch 402, two powershelves (e.g., a power shelf 404A and a power shelf 404B, collectivelycalled the “power shelves 404”), and a bus bar 406. In some embodiments,rack column 400 includes a separation bar 410 and rack-mountedcomponents such as servers 412. The rack-mounted components provideserver functionalities such as servicing requests and supportingapplications and services. The separation bar isolates regions withinthe rack column. Here, a top half of the rack column is separated from abottom half of the rack column by the separation bar. The number andplacement of rack-mounted components 412 is merely exemplary and notintended to be limiting. In some embodiments, power shelf 404A powersrack-mounted components 412 and power shelf 404B powers a different setof rack-mounted components, which are not shown here for simplicity.

Computer network switch 402 (also known as a top-of-rack “TOR” switch)is configured to provide control of one or more rack columns. Computernetwork switch 402 can be configured to manage components within arespective rack column. For example, computer network switch canimplement the power management processes described here by notifyingservers connected to the computer network switch of an imminent powerloss. This allows the servers to prepare for a loss in power. Computernetwork switch 402 is configured to interconnect rack columns. Forexample, the computer network switch facilitates communications betweenrack columns. One example of a TOR switch is a Facebook® Wedge switch,which is an OS-agnostic switch that includes a server module. Componentssuch as the server module in the Wedge switch can be easily modified andreplaced according to networking needs and when new technology becomesavailable. Although shown at the top of the rack column, computernetwork switch 402 may be disposed elsewhere (middle, bottom, etc.) inthe rack column in other embodiments.

In some embodiments, computer network switch 402 includes a baseboardmanagement controller (BMC). A BMC is a specialized controller embeddedin servers. In some embodiments, the BMC is implemented by asystem-on-chip (SoC) with its own CPU, memory, storage, and IO. A BMCcan be coupled to sensors to read environmental conditions, and can becoupled to fans to control temperature. The BMC is configured to provideother system management functions such as remote power control, serialover LAN, monitoring and error logging of the server host CPU andmemory, and the like. An example of a BMC is OpenBMC.

Rack column 400 includes two power shelves 404. Each of the powershelves includes three PSU and BBU pairs. A PSU is configured tocommunicate with and manage one or more corresponding BBUs. For example,the PSU can instruct the BBU to charge or discharge. The PSU can sendmessages to computer network switch 402 regarding the PSU itself or theBBU associated with the PSU. Each PSU may include a power supply circuit(e.g., the power supply circuit 300 of FIG. 3). Each of the powershelves 404 establishes its own power zone (e.g., the power shelf 404Apowers a first power zone above separation bar 410 and the power shelf404B powers a second power zone below the separation bar 410). Forexample, the power shelf 404A can provide power to the first power zonevia DC bus bar 406. One or more DC bus bars can be coupled in parallelto the power supply circuits of the PSUs. The DC bus bars 406 can bundletwo wires (positive and negative terminations, with negative chassispre-mating bus bar) providing a DC differential voltage (e.g., 12.5V).The two power zones are electrically isolated from each other. In someembodiments, there may be a provision to connect the power zones in therear of the rack making up one power zone. In this case, only one powershelf would be used per rack column for a low-power implementation.

In the embodiment of the rack column 400 shown in FIG. 4, the powershelves 404 are positioned in the middle of their respective powerzones. One set of the DC bus bars 406 extends above the power shelves404 and another set extends below the power shelves. This positioningadvantageously reduces distribution conduction losses from the DC busbars 406 by minimizing the length of each set of the DC bus bars 406 andby splitting the current in half.

The rack column 400 may be implemented in accordance with the OpenCompute Project standard (Open Rack V2) having a rack height of around87 inches or 42 open rack units. In the Open Compute Project standard,an open rack unit or “OU” is a unit for the height of a chassis mountedin the rack. 1 OU is 48 mm high. For example, each of the power shelves404 can occupy 3 OU of space.

Power input equipment, e.g., a top power rail (e.g., a rack powerdistribution unit corresponding to rack PDU 116 of FIG. 1) can take up 1OU to 3 OU of space in the rack column 400. In the example shown here,computer network switch 402 takes up 1 OU of space. For example, thepower zone corresponding to power shelf 404A provides 18 OUs of space,including 8 OUs of space above the power shelf 404A, 8 OU of space belowthe power shelf 404A, and 2 OU of space taken up by the power shelf404A. The two power zones can be separated by a separation bar 410 thattakes up 0.5 OU of space. The separation bar can serve as a stabilizerand a calibrator. For example, the separation bar can be 339 mm long. DCbus bar 406 can be coupled to the power input equipment and the powershelf, to deliver power from the rack PDU to the power shelf.

Three-phase AC power is delivered to power shelves 404, and thendistributed to the three PDUs within that shelf. In some embodiments,there is a one-to-one relationship between the PSU and the BBU locatedbelow it. The PSU is only responsible for charging the BBU below it.When there is an AC outage, the PSU will use the power from the BBU andconvert it to the DC voltage, e.g., 12.5V, used by the componentsserviced by the power shelf.

In some embodiments, computer network switch 402 is configured tomonitor the health of power components within one or more rack columns.For example, the computer network switch may include a rack monitor formonitoring. The monitoring also reveals whether there is an AC outagethat impacts a power shelf. The topology of the rack column and powerdelivery system can be used for a notification system (sometimes calleda power loss siren) to allow services to automate a quick response to anoutage. An example of a rack monitor is shown in FIG. 5. In someembodiments, a rack monitor may be mounted on top of the computernetwork switch 402. In some embodiments, the rack monitor may beimplemented by a microcontroller provided inside computer network switch402 to help implement the power loss siren technique described here.

In operation, a PSU senses an AC outage. In some embodiments, a PSUincludes a sensor that detects an AC outage, for example when there isno current. In some embodiments, a computer network switch 402 monitorsthe health of power components within one or more rack columns byreceiving sensor readings or other notifications from the PSU. The PSUsignals to the BBU that an AC outage has occurred, causing the BBU todischarge and provide power to the PSU. For example, according to theOpen Compute Project standard, BBUs are configured to provide 90 secondsof back-up power. According to the power management techniques describedhere, a PSU detects an outage, sends a message regarding an AC networkoutage. In some embodiments, the message is received by a computernetwork switch such as a baseboard management controller (BMC) in theswitch. The computer network switch then broadcasts the outageinformation to associated rack-mounted components such as servers,allowing the servers to prepare for power loss. The rack-mountedcomponents have time to wind down processes before they perceive a powerloss, e.g., before the BBUs cease providing power to the PSUs.

Conventionally, servers are not notified of an imminent power loss, andthus are impacted if they are unable to wind down processes during thetime that back-up power is provided by the BBUs to the servers (e.g., 90seconds). According to an embodiment of the present disclosure, theservers are notified about an imminent loss of power before or duringthe time that BBUs (rather than the AC input) are providing power to thePSUs. This allows servers to proactively respond and prepare for powerloss, e.g., store data, and reduces the impact to the services.

The following figure is an example of a rack monitor that is configuredto monitor for a power outage and notify a computer network switch of animminent power loss.

FIG. 5 is a block diagram illustrating an embodiment of a rack monitorfor providing a power loss siren. The rack monitor may be providedinside a computer network switch such as switch 402. The rack monitormay be a stand-alone device with its own chassis occupying 1 OU or lessof space in a rack column.

The rack monitor 500 includes a main board 510 and an input-output (I/O)board 550. In various embodiments, the rack monitor is configured toprovide power monitoring functions such as monitoring power loss to oneor more PSUs. For example, the PSUs may have sensors detecting flow ofcurrent and report readings or abnormalities to the rack monitor. Insome embodiments, the rack monitor is configured to monitor 18 or morepower supplies, e.g., via serial interface 556 such as RS485communication from the power shelves. One rack monitor may be used formonitoring several rack columns. PSUs report their status to the rackmonitor via software running on the monitor. The rack monitor acts as aMaster, and all connected devices (e.g., PSUs) act as Slaves. An exampleof a physical rack monitor unit is shown in FIGS. 7A and 7B.

Main board 510 includes memory 512 and microserver chip 514. The mainboard can be implemented by a microserver card. Main board 510 receivesstatus signals from other devices. For example, signals received by theGPIO ports 554A and 554B of the I/O board 550 are analyzed to determinehow to respond to various fail signals. Main board 510 is configured toimplement the processes described here, e.g., the one shown in FIG. 6.For example, memory 512 stores instructions for microserver chip 514 toexecute the processes described here. In some embodiments, memory 512may store log data and forward the log data to a remote database inbatches. The rack monitor reads the state of health of IT equipment forwhich the monitor is responsible and reports discovered failures. Forexample, when a power outage occurs, the main board determines a waitperiod, and, at the end of the wait period, notifies a computer networkswitch or hosts of a rack of an imminent power loss.

I/O board 550 includes a baseboard management controller (BMC) 552, aphysical chip 558, and DC to DC converter 560. The I/O board isconfigured to provide ports, serial communication, and out of bandmanagement functionalities to the main board 510. The I/O board iscommunicatively coupled to the main board 510 via the BMC 552. Forexample, the I/O board is coupled to the main board via an Ethernetsystem managed by physical layer chip 558, which Ethernet system can beconnected to a top-of-rack switch. GPIO ports (here, a first GPIO portis 554A and a second GPIO port is 554B, and collectively they are calledGPIO ports 554) monitor fail signals from other IT/power equipment. Forexample, power outage messages can be received via the GPIO ports 554.Each IT/power equipment device may have a dedicated GPIO for reportingits failures. For example, when a power shelf is operational, two pinsare open, and they are shorted when a failure occurs.

The rack monitor may be powered by an auxiliary power such as the oneshown in FIG. 2. The DC/DC converter 560 steps the voltage down toappropriate voltages for the rack monitor components.

The following figure is an example of a process that can be carried outby the rack monitor to provide a power loss siren.

FIG. 6 is a flow chart illustrating an embodiment of a process forproviding a power loss siren. A power loss siren is a messagedistributed to rack components such as servers to warn the componentsthat they are about to experience a power loss.

A power interruption to the power supply is detected (602). For example,a sensor on a PSU detects that there is a power outage. The sensor maymeasure current to determine whether power is being delivered to the PSUor monitor other aspects of performance and power health. In someembodiments, a computer network switch (e.g., a rack monitor in anetwork switch) coordinates or monitors PSUs.

A message is provided regarding the detected power interruption (604).The message may be provided to a component such as a computer networkswitch. In some embodiments, a network switch includes a baseboardmanagement controller (BMC) configured to receive the message. In someembodiments, the general-purpose input/output (GPIO) signal indicating apower loss condition. For example, the message created as a result ofthe GPIO notification includes an outage state, a duration, or a port.The outage state indicates a condition, for example that there is apower outage, as indicated by a flag, number, or the like.

In some embodiments, the condition indicates that the power loss is aredundancy loss. A redundancy loss refers to loss of power to aredundant PSU. Referring to FIG. 4, one of the three pairs of PSUs andBBUs in power shelf 404A is redundant, meaning that voltage and powersupplied through the DC bus is sufficient to power the rack-mountedequipment even when one of the multiple pairs is inoperative. Theredundant inoperative pair may indicate a power loss. In someembodiments, in this situation a notification is not sent becauseinterested services will be able to perform as usual with the remainingtwo PSU/BBU pairs.

The duration indicates how much time has elapsed since the beginning ofthe AC outage or a countdown to when BBU power will run out. Forexample, if the sending of the message is delayed for 45 seconds and aBBU provides 90 seconds of backup power, then the countdown willindicate 45 more seconds until power is lost. The port indicates wherean outage signal is received from or where an outage is reported from.

The message can be output, stored, or used in a variety of ways. Forexample, in response to the message, a computer network switch notifiesservers connected to the computer network switch, allowing the serversto prepare for a loss of power (606). In some embodiments, the computernetwork switch notifies all hosts within a rack of the imminent powerloss. A siren client library on the hosts delivers notifications tointerested services.

The message, which indicates an imminent power loss event can be loggedand tracked in a remote database. This message is sometimes called a“siren” or “notification.” For example a data center wide powermanagement system that manages the power hierarchy can log the messageand use it for analytics. Facebook® Dynamo is an example of a datacenter wide power management system. One component of Dynamo is a Dynamoagent that is deployed on each server in the data center. The Dynamoagent reads power, executes power capping/uncapping, commands, andcommunicates with Dynamo controllers. In some embodiments, Dynamo agentscommunicate with each other via Dynamo controllers. A Dynamo controllerruns on a group of dedicated servers, monitors data from the Dynamoagents under their control, and helps to protect data center powerdevices. A hierarchy of Dynamo controllers mirrors the topology of thedata center's power hierarchy to monitor and control downstream serversfor that device.

In some embodiments, the message is sent to the computer network switchat the same time that another message instructing the BBUs to dischargeis sent to the BBUs. In some embodiments, the message is sent to thecomputer network switch after a waiting period. For example, instead ofsending the outage message immediately upon detection of an AC outage, apredefined waiting period elapses before sending the message. Thewaiting period can be selected to reduce false positives. For example,for a BBU that provides backup power for 90 seconds, 45 seconds can bethe waiting period. That is, servers are notified around 45 secondsafter a loss of power instead of right away to account for shortinterruptions to power that are not actual prolonged AC outages.

Power to a PSU can be briefly cut. For example, sometimes backup power(e.g., a generator or reserve power) is engaged, and the BBUs do notneed to discharge to power the PSUs because the PSUs will be powered bythe standby generators. For example, as described with respect to FIG.1, each main switchboard may have a generator, which provides backuppower in the event of an outage. Typically, backup power is engagedwithin around 30 seconds. As another example, there may be a brief (lessthan the timeout period) outage due to causes other than a utility ACoutage. For example, maintenance activities that are not true utility ACoutages involve AC power disturbances or outages. For example, a testmay cut AC power to the racks for less than around 45 seconds. By thetime the timeout period has elapsed, an imminent power loss is expectedbecause the BBUs will only last a short while longer (e.g., 45 moreseconds for a total of 90 seconds). Typically, if power is not providedto the PSU within the waiting period, then there is a true AC outage(e.g., a standby generator is not functioning).

In some embodiments, the power loss notification is secured to preventforgery. For example, power loss notifications can be cryptographicallysigned by a rack switch with a signing key accessible only to the rackswitch. A client library verifies the signature and ignoresnotifications that do not have a valid signature.

A server can respond to a power outage in a variety of ways. Eachservice can respond differently to the imminent power loss. An exampleof how a server can prepare for a power outage at the database levelwill now be described. In a typical MySQL arrangement, a master controlsa number of slaves. If a power outage affects a master, the slaves willtypically not know what to do. Thus, to prepare for an outage, a slavethat will not be affected by the power outage can be promoted to bemaster. This is called a “live promotion.” When the server receives asignal of an imminent outage, the server determines a slave that isoutside the region of outage to promote to master, and promotes theslave to master. In a typical configuration, a redundant device such asa load balancer unit (LBU) stores data such as state information that isused when a slave is promoted to master.

One example of how a tier might respond to imminent power loss has beendescribed. Other tiers can respond differently. For example, a haltcommand can be executed, which stops execution and gracefully shuts downoperations while storing in process data to memory so that the powerloss does not impact the service.

Interested services can find out if its rack is about to lose power invariety of ways. First, an application can listen for a siren alertdirectly. For example, an application of the service listens to a portfor power loss siren alerts in the form of UDP packets. The power losssiren alert can be UTF-8 JSON encoded thrift objects. The object can beverified to determine that it is an authentic siren using acryptographic signature of the rack switch. Second, an interestedapplication can make calls using a provided library to watch for sirenalerts. Third, a local daemon on a server rack can push siren alerts tosubscribers. The local daemon is a siren proxy that can pushJSON-encoded power loss sirens to standard HTTP2 streaming clients.

FIG. 7A is a front view of an example of a physical rack monitor. FIG.7B is a perspective view of an example of a physical rack monitor. Thephysical rack monitor shown here includes a chassis with opening forGPIO connections, rack connections, and a connection to TOR. Thephysical rack monitor shown here corresponds to the block diagram shownin FIG. 5. FIG. 8 shows an example of a physical rack monitor mounted ona server rack. In this example, the rack monitor is mounted with a TORcomputer network switch.

The power management system disclosed here improves the functioning of acomputer network, specifically one or more server racks by detecting anAC outage and notifying servers of imminent power loss to enable theservers to prepare for the power loss and minimize serviceinterruptions. In another aspect, in some embodiments, the power lossdetection and notification is entirely performed by internal componentsof a server rack and does not rely on external services to notifyregarding a power loss.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A power management system, comprising: adetection unit configured to detect a power interruption to a powersupply; and a communication interface configured to, in response to thedetected power interruption, provide a message regarding the detectedpower interruption; wherein, in response to the detected powerinterruption, a computer network switch provides notifications to aplurality of servers connected to the switch to allow the plurality ofservers to prepare for a loss in power.
 2. The system of claim 1,wherein the message is provided after a predefined waiting period. 3.The system of claim 1, wherein the message is a general-purposeinput/output (GPIO) signal.
 4. The system of claim 1, wherein themessage includes an outage state, duration, and port of origin of themessage.
 5. The system of claim 1, wherein data of at least one of theservers is written to a persistent data storage device in response toreceiving at least one of the notifications.
 6. The system of claim 1,wherein the power management system is installed in a server rack. 7.The system of claim 1, further comprising a memory configured to storelog data associated with the power interruption to the power supply. 8.The system of claim 1, wherein system is further configured to: collectlog data associated with the power interruption to the power supply; andforward the log data to a remote database.
 9. The system of claim 1,wherein the detection unit is configured to monitor a plurality of powersupplies.
 10. The system of claim 1, wherein the message includes ageneral-purpose input-output (GPIO) signal, the signal including astatus indication and a specification of a time duration.
 11. The systemof claim 1, wherein the message identifies which power supply among aplurality of power supplies has lost power.
 12. The system of claim 11,wherein the power supply that has lost power is a redundant powersupply.
 13. The system of claim 1, wherein the detection of the powerinterruption to the power supply is based on a duration encoded in asignal, the duration indicating at least one of: a beginning time of thepower interruption and a countdown to when backup power will stopproviding power.
 14. The system of claim 1, wherein the notificationsprovided by the network switch are authenticable by a cryptographicsignature of the power management system from which the notificationsoriginated.
 15. The system of claim 1, wherein in response to at leastone of the provided notifications, at least one of the plurality ofservers prepare for the loss in power by initiating a promotion of aselected slave to a master and the selected slave is determined to benot subject to the power interruption.
 16. The system of claim 1,wherein in response to the provided notification, the message wasprovided to a baseboard management controller (BMC) included in thecomputer network switch.
 17. The system of claim 1, wherein thenotifications are provided to the plurality of servers at substantiallythe same time that an instruction is sent to a backup power supply todischarge.
 18. The system of claim 1, wherein the notifications areprovided to the plurality of servers after a threshold delay periodafter sending an instruction to a backup power supply to discharge. 19.A method comprising: detecting a power interruption to a power supply;and in response to the detected power interruption, providing a messageregarding the detected power interruption, wherein in response to thedetected power interruption, a computer network switch providesnotifications to a plurality of servers connected to the switch to allowthe plurality of servers to prepare for a loss in power.
 20. A computerprogram product embodied in a non-transitory computer readable storagemedium and comprising computer instructions for: detecting a powerinterruption to a power supply; and in response to the detected powerinterruption, providing a message regarding the detected powerinterruption, wherein in response to the detected power interruption, acomputer network switch provides notifications to a plurality of serversconnected to the switch to allow the plurality of servers to prepare fora loss in power.